A. Field of the Invention
The present invention relates generally to content retrieval on the world wide web, and more particularly, to automated web crawling.
B. Description of the Related Art
The World Wide Web (“web”) contains a vast amount of information. Search engines assist users in locating desired portions of this information by cataloging web pages. Typically, in response to a user's request, the search engine returns references to documents relevant to the request.
Search engines may base their determination of the user's interest on search terms (called a search query) entered by the user. The goal of the search engine is to identify links to high quality relevant results based on the search query. Typically, the search engine accomplishes this by matching the terms in the search query to a corpus of pre-stored web documents. Web documents that contain the user's search terms are considered “hits” and are returned to the user.
The corpus of pre-stored web documents may be stored by the search engine as an index of terms found in the web pages. Documents that are to be added to the index may be automatically located by a program, sometimes referred to as a “spider,” that automatically traverses (“crawls”) web documents based on the uniform resource locators (URLs) contained in the web documents. Thus, for example, a spider program may, starting at a given web page, download the page, index the page, and gather all the URLs present in the page. The spider program may then repeat this process for the web pages referred to by the URLs. In this way, the spider program “crawls” the world wide web based on its link structure.
Some web sites generate pages dynamically based on parameters specified in the URLs. Some of these parameters may be redundant and may not affect the content returned by the web site. This can result in many different URLs that all refer to the same content. Different URLs that refer to the same content can cause the spider program to redundantly index the web page. This is undesirable, as it can, for example, lead to a decrease in the quality of the operation of the search engine, can affect the feasibility of crawling particular sites, and can significantly increase the cost and time involved in crawling.
Thus, there is a need in the art to more effectively crawl web sites that use different URLs that refer to the same content.